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(57) Abstract 



A network system includes a client computer that can present multimedia content to a user and an annotation server that can transmit, 
to the client computer, annotations for the multimedia content. Users can be presented with annotations generated by other users and can 
create their own annotations that can be viewed by others. Users can be notified of new annotations by an electronic mail (email) message, 
can create new annotations by email messages, and can easily access a temporal segment of multimedia content corresponding to a new 
annotation based on information in email messages. 
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ANNOTATION CREATION AND NOTIFICATION VIA ELECTRONIC MAIL 

A portion of the disclosure of this patent document contains material which 
5 is subject to copyright protection. The copyright owner has no objection to the 
facsimile reproduction by anyone of the patent document or the patent disclosure, as 
it appears in the Patent and Trademark Office patent file or records, but otherwise 
reserves all copyright rights whatsoever. 

10 RELATED APPLICATIONS 

This application claims priority to U.S. Provisional Application No. 
60/100,452, filed September 15, 1998, entitled "Annotations for Streaming Video 
on the Web: System Design and Usage", to Anoop Gupta and David M. Bargeron. 

15 TECHNICAL FIELD 

This invention relates to networked client/server systems and to methods of 
annotating multimedia content in such systems. More particularly, the invention 
relates to creation and notification of new annotations via electronic mail. 

20 BACKGROUND OF THE INVENTION 

The advent of computers and their continued technological advancement has 
revolutionized the manner in which people work and live. One such advancement 
is that of multimedia presentations, where presentations that used to be made in 
person (e.g., college lectures, workplace training sessions, etc.) can be provided to a 
25 computer user as multimedia data (e.g., video, audio, text, and/or animation data). 
Today, such presentations are primarily video and audio, but a richer, broader 
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digital media era is emerging. Multimedia presentations provide many benefits, 
such as allowing the presentation data to be created at a single time yet be presented 
to different users at different times and in different locations throughout the world. 

These multimedia presentations are provided to a user as synchronized 
5 media. Synchronized media refers to multiple media objects that share a common 
timeline. Video and audio are examples of synchronized media - each is a separate 
data stream with its own data structure, but the two data streams are played back in 
synchronization with each other. Virtually any media type can have a timeline. For 
example, an image object can change like an animated .gif file, text can change and 

10 move, and animation and digital effects can happen over time. This concept of 
synchronizing multiple media types is gaining greater meaning and currency with 
the emergence of more sophisticated media composition frameworks implied by 
MPEG-4, Dynamic HTML, and other media playback environments. 

The term "streaming" is used to indicate that the data representing the 

15 various media types is provided over a network to a client computer on a real-time, 
as-needed basis, rather than being pre-delivered in its entirety before playback. 
Thus, the client computer renders streaming data as it is received from a network 
server, rather than waiting for an entire "file" to be delivered. 

Multimedia presentations may also include "annotations" relating to the 

20 multimedia presentation. An annotation is data (e.g., audio, text, video, etc.) that 
corresponds to a multimedia presentation. Annotations can be added by anyone 
with appropriate access rights to the annotation system. These annotations typically 
correspond to a particular temporal location in the multimedia presentation and can 
provide a replacement for much of the "in-person" interaction or "classroom 

25 discussion" that is lost when the presentation is not made "in-person" or "live". As 
part of an annotation, a user can comment on a particular point, to which another 
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user (or lecturer/presenter) can respond in a subsequent annotation. This process 
can continue, allowing a "classroom discussion" to occur via these annotations. 

However, in order to gain wide range acceptance of multimedia 
presentations and the use of annotations with such presentations, a user-friendly 
5 manner for interacting with the presentations should be available. 

Additionally, it would be useful to be able to notify individuals when new 
annotations are added. In some systems, the author or other "owner" of a 
multimedia presentation can be notified by electronic mail (email) whenever 
someone adds a new annotation to the presentation. However, there are several 
10 problems with such systems. One such problem is that the author or creator of the 
annotation has no control over (or perhaps even knowledge of) who is being 
notified of the new annotation. Another problem is that in order for the 
author/owner of the presentation to view the annotation or to access the multimedia 
presentation, the author/owner must manually access the multimedia presentation 
1 5 and manually locate the portion corresponding to the new annotation. 

The invention described below addresses these and other disadvantages of 
current annotations systems, facilitating annotation creation and notification via 
email. 

20 SUMMARY OF THE INVENTION 

A network system includes a client computer that can present multimedia 
content to a user and an annotation server that can transmit, to the client computer, 
annotations for the multimedia content. Users are presented with annotations 
generated by other users and can create their own annotations that can be presented 
25 to others. Users can be notified of new annotations by electronic mail (email), and 
can also create new annotations by email. 
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According to one aspect of the invention, a user creating a new annotation 
can identify one or more other users that are to be notified, by email, of the new 
annotation. Users to be notified by email can be identified manually by the 
annotation author, or alternatively a default user(s) to be notified by email can be 
5 associated with the media content. Additionally, group identifiers can be used to 
identify groups of users to be notified of the new annotation by email. 

According to another aspect of the invention, an email message notifying a 
recipient of a new annotation also includes a user-selectable link for the temporal 
segment of multimedia content corresponding to the annotation. Upon receipt of 
10 the email message, the recipient can select the link and have the media server 
present, to the user, the temporal segment of multimedia content that corresponds to 
the annotation. 

According to another aspect of the invention, a recipient of an email message 
notifying the recipient of a new annotation can create another new annotation by 

15 replying to the email message. The recipient can enter the new annotation content 
and select the "reply" option from his or her electronic mail application. Selection 
of the "reply" option sends the reply email message to the annotation server, which 
creates another new annotation using the content entered by the recipient. 

According to another aspect of the invention, a new email message can be 

20 generated that includes the content for a new annotation but that is not in response 
to an email notification of a new annotation. This new email message includes the 
content for the new annotation as well as information that can be used by the 
annotation server to create a new annotation. Examples of such information include 
the multimedia content to which the new annotation corresponds, and the temporal 

25 range of the media content to which the new annotation corresponds. 
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According to another aspect of the invention, one or more annotations can be 
created by forwarding an email thread to the annotation server. Upon receiving an 
email thread, including multiple email messages that were forwarded or replied to 
by various users, the annotation server creates a new annotation including all of 
5 these email messages. Alternatively, a separate annotation for each email message 
in the thread may be created. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example and not limitation in 
1 0 the figures of the accompanying drawings. The same numbers are used throughout 
the figures to reference like components and/or features. 

Fig. 1 shows a client/server network system and environment in accordance 
with one embodiment of the invention. 

Fig. 2 shows a general example of a computer that can be used as a client or 
1 5 server in accordance with the invention. 

Fig. 3 illustrates an exemplary annotation server and a client computer in 
more detail. 

Fig. 4 shows an exemplary structure for an annotation entry that is 
maintained by an annotation server of Fig. 3. 
20 Fig. 5 illustrates implicit annotation collections for annotations maintained 

by an annotation server of Fig. 3. 

Fig. 6 illustrates an annotation toolbar in accordance with one embodiment 
of the invention. 

Fig. 7 shows an "add new annotation" dialog box that allows a user to create 
25 a new annotation. 

Fig. 8 shows an example email message notification of a new annotation. 
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Fig. 9 shows an example email reply message including a new annotation. 

Fig. 10 is a flowchart illustrating exemplary steps in accessing media content 
from an email message in accordance with the invention. 

Fig. 11 is an exemplary diagram illustrating the creation of an email thread. 
5 Fig. 12 is a flowchart illustrating an exemplary process in adding new 

annotations from an email message in accordance with the invention. 

Fig. 13 is a diagrammatic illustration of a graphical user interface window 
displaying annotations and corresponding media segments concurrently in 
accordance with the invention. 

10 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

General Network Structure 

Fig. 1 shows a client/server network system and environment in accordance 
with one embodiment of the invention. Generally, the system includes multiple 

15 network server computers 10, 11, 12, and 13, and multiple (/?) network client 
computers 15. The computers communicate with each other over a data 
communications network. The communications network in Fig. 1 comprises a 
public network 16 such as the Internet. The data communications network might 
also include, either in addition to or in place of the Internet, local-area networks 

20 and/or private wide-area networks. 

Streaming media server computer 1 1 has access to streaming media content 
in the form of different media streams. These media streams can be individual 
media streams (e.g., audio, video, graphical, etc.), or alternatively can be composite 
media streams including two or more of such individual streams. Some media 

25 streams might be stored as files in a database or other file storage system, while 
other media streams might be supplied to the server on a "live" basis from other 
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data source components through dedicated communications channels or through the 
Internet itself 

There are various standards for streaming media content and composite 
media streams. The "Advanced Streaming Format" (ASF) is an example of such a 
5 standard, including both accepted versions of the standard and proposed standards 
for future adoption. ASF specifies the way in which multimedia content is stored, 
streamed, and presented by the tools, servers, and clients of various multimedia 
vendors. Further details about ASF are available from Microsoft Corporation of 
Redmond, Washington. 

10 Annotation server 10 controls the storage of annotations and their provision 

to client computers 15. The annotation server 10 manages the annotation meta data 
store 18 and the annotation content store 17. The annotation server 10 
communicates with the client computers 15 via any of a wide variety of known 
protocols, such as the Hypertext Transfer Protocol (HTTP). The annotation server 

15 10 can receive and provide annotations via direct contact with a client computer 15, 
or alternatively via electronic mail (email) via email server 13. The annotation 
server 10 similarly communicates with the email server 13 via any of a wide variety 
of known protocols, such as the Simple Mail Transfer Protocol (SMTP). 

The annotations managed by annotation server 10 correspond to the 

20 streaming media available from media server computer 11. In the discussions to 
follow, the annotations are discussed as corresponding to streaming media. 
However, it should be noted that the annotations can similarly correspond to "pre- 
delivered" rather than - streaming media, such as media previously stored at the 
client computers 15 via the network 16, via removable magnetic or optical disks, 

25 etc. 
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When a user of a client computer 15 accesses a web page containing 
streaming media, a conventional web browser of the client computer 15 contacts the 
web server 12 to request a Hypertext Markup Language (HTML) page. The client- 
based browser also submits requests to the media server 11 for streaming data, and 
5 the annotation server 10 for any annotations associated with the streaming data. 
When a user of a client computer 1 5 desires to add or retrieve annotations, the client 
computer 15 contacts the annotation server 10 to perform the desired 
addition/retrieval. 

10 Exemplary Computer Environment 

In the discussion below, the invention will be described in the general 
context of computer-executable instructions, such as program modules, being 
executed by one or more conventional personal computers. Generally, program 
modules include routines, programs, objects, components, data structures, etc. that 

15 perform particular tasks or implement particular abstract data types. Moreover, 
those skilled in the art will appreciate that the invention may be practiced with other 
computer system configurations, including hand-held devices, multiprocessor 
systems, microprocessor-based or programmable consumer electronics, network 
PCs, minicomputers, mainframe computers, and the like. In a distributed computer 

20 environment, program modules may be located in both local and remote memory 
storage devices. 

Fig. 2 shows a general example of a computer 20 that can be used as a client 
or server in accordance with the invention. Computer 20 is shown as an example of 
a computer that can perform the functions of any of server computers 10-13 or a 
25 client computer 15 of Figure 1. 
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Computer 20 includes one or more processors or processing units 21, a 
system memory 22, and a bus 23 that couples various system components including 
the system memory 22 to processors 2 1 . 

The bus 23 represents one or more of any of several types of bus structures, 
5 including a memory bus or memory controller, a peripheral bus, an accelerated 
graphics port, and a processor or local bus using any of a variety of bus 
architectures. The system memory includes read only memory (ROM) 24 and 
random access memory (RAM) 25. A basic input/output system (BIOS) 26, 
containing the basic routines that help to transfer information between elements 

10 within computer 20, such as during start-up, is stored in ROM 24. Computer 20 
further includes a hard disk drive 27 for reading from and writing to a hard disk, not 
shown, a magnetic disk drive 28 for reading from and writing to a removable 
magnetic disk 29, and an optical disk drive 30 for reading from or writing to a 
removable optical disk 31 such as a CD ROM or other optical media. The hard disk 

15 drive 27, magnetic disk drive 28, and optical disk drive 30 are connected to the 
system bus 23 by an SCSI interface 32 or some other appropriate interface. The 
drives and their associated computer-readable media provide nonvolatile storage of 
computer readable instructions, data structures, program modules and other data for 
computer 20. Although the exemplary environment described herein employs a 

20 hard disk, a removable magnetic disk 29 and a removable optical disk 31, it should 
be appreciated by those skilled in the art that other types of computer readable 
media which can store data that is accessible by a computer, such as magnetic 
cassettes, flash memory cards, digital video disks, random access memories 
(RAMs) read only memories (ROM), and the like, may also be used in the 

25 exemplary operating environment. 
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A number of program modules may be stored on the hard disk, magnetic disk 
29, optical disk 31, ROM 24, or RAM 25, including an operating system 35, one or 
more application programs 36, other program modules 37, and program data 38. A 
user may enter commands and information into computer 20 through input devices 
5 such as keyboard 40 and pointing device 42. Other input devices (not shown) may 
include a microphone, joystick, game pad, satellite dish, scanner, or the like. These 
and other input devices are connected to the processing unit 21 through an interface 
46 that is coupled to the system bus. A monitor 47 or other type of display device is 
also connected to the system bus 23 via an interface, such as a video adapter 48. In 

10 addition to the monitor, personal computers typically include other peripheral 
output devices (not shown) such as speakers and printers. 

Computer 20 operates in a networked environment using logical connections 
to one or more remote computers, such as a remote computer 49. The remote 
computer 49 may be another personal computer, a server, a router, a network PC, a 

1 5 peer device or other common network node, and typically includes many or all of 
the elements described above relative to computer 20, although only a memory 
storage device 50 has been illustrated in Fig. 2. The logical connections depicted in 
Fig. 2 include a local area network (LAN) 51 and a wide area network (WAN) 52. 
Such networking environments are commonplace in offices, enterprise-wide 

20 computer networks, intranets, and the Internet. In the described embodiment of the 
invention, remote computer 49 executes an Internet Web browser program such as 
the "Internet Explorer" Web browser manufactured and distributed by Microsoft 
Corporation of Redmond, Washington. 

When used in a LAN networking environment, computer 20 is connected to 

25 the local network 51 through a network interface or adapter 53. When used in a 
WAN networking environment, computer 20 typically includes a modem 54 or 



BNSDOCID: <WO 001 6541 A1J_> 



WO 00/1 654 1 PCT/US99/2 1 344 

11 

other means for establishing communications over the wide area network 52, such 
as the Internet. The modem 54, which may be internal or external, is connected to 
the system bus 23 via a serial port interface 33. In a networked environment, 
program modules depicted relative to the personal computer 20, or portions thereof, 
5 may be stored in the remote memory storage device. It will be appreciated that the 
network connections shown are exemplary and other means of establishing a 
communications link between the computers may be used. 

Generally, the data processors of computer 20 are programmed by means of 
instructions stored at different times in the various computer-readable storage media 

10 of the computer. Programs and operating systems are typically distributed, for 
example, on floppy disks or CD-ROMs. From there, they are installed or loaded 
into the secondary memory of a computer. At execution, they are loaded at least 
partially into the computer's primary electronic memory. The invention described 
herein includes these and other various types of computer-readable storage media 

15 when such media contain instructions or programs for implementing the steps 
described below in conjunction with a microprocessor or other data processor. The 
invention also includes the computer itself when programmed according to the 
methods and techniques described below. Furthermore, certain sub-components of 
the computer may be programmed to perform the functions and steps described 

20 below. The invention includes such sub-components when they are programmed as 
described. In addition, the invention described herein includes data structures, 
described below, as embodied on various types of memory media. 

For purposes of illustration, programs and other executable program 
components such as the operating system are illustrated herein as discrete blocks, 

25 although it is recognized that such programs and components reside at various times 
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in different storage components of the computer, and are executed by the data 
processor(s) of the computer. 

Client/Server Relationship 

5 Fig. 3 illustrates an exemplary annotation server and a client computer in 

more detail. As noted above, generally, commands are formulated at client 
computer 15 and forwarded to annotation server 10 via HTTP requests. In the 
illustrated embodiment of Fig. 3, communication between client 15 and server 10 is 
performed via HTTP, using commands encoded as Uniform Resource Locators 
10 (URLs) and data formatted as object linking and embedding (OLE) structured 
storage documents, or alternatively using Extensible Markup Language (XML). 

Client 15 includes an HTTP services (HttpSvcs) module 152, which 
manages communication with server 10, and an annotation back end (ABE) module 
151, which translates user actions into commands destined for server 10. A user 
1 5 interface (MMA) module 1 50 provides the user interface (UI) for a user to add and 
select different annotations and be presented with the annotations. According to 
one implementation, the user interface module 150 supports ActiveX controls that 
display an annotation interface for streaming video on the Web. 

Client 15 also includes a web browser module 153, which provides a 
20 conventional web browsing interface and capabilities for the user to access various 
servers via network 16 of Fig. 1. Web browser 153 also provides the interface for a 
user to select particular media streams for presentation. The user can select which 
one of different versions of multimedia content he or she wishes to receive from 
media server 1 1 of Fig. 1 . This selection can be direct (e.g., entry of a particular 
25 URL or selection of a "low resolution" option), or indirect (e.g., entry of a 
particular desired playback duration or an indication of system capabilities, such as 
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"slow system" or "fast system"). Alternatively, other media presentation interfaces 
could be used. 

Client 15 further includes an electronic mail (email) agent 154 and an email 
interface 155. Email agent 154 manages the generation, transmission, receipt, and 
5 storage of email messages at client 15. Email interface 155 provides the interface 
for a user to make use of the email services of email agent 1 54, including creating, 
modifying, deleting, responding to, etc. email messages. Email messages can be 
received by client 15 from other client computers or servers either directly or 
indirectly via an email server (e.g., email server 13 of Fig. 1). Additionally, new 

10 email messages, including replies to received email messages, can be forwarded to 
other client computers or servers on network 16 either directly or indirectly via an 
email server (e.g., email server 13). Any of a wide variety of conventional 
electronic mail protocols can be used by email agent 154 to send and receive email 
messages, such as SMTP. 

15 Annotation server 10 includes the Multimedia Annotation Web Server 

(MAWS) module 130, which is an Internet Services Application Programming 
Interface (ISAPI) plug-in for Internet Information Server (IIS) module 135. 
Together, these two modules provide the web server functionality of annotation 
server 10. Annotation server 10 also includes an HTTP Services module 131 which 

20 manages communication with client 15. In addition, annotation server 10 utilizes 
the Windows Messaging Subsystem 134 to facilitate communication with email 
server 13 of Fig. 1, and an email reply server 133 for processing incoming email 
received from email server 13. 

Annotation server 10 further includes an annotation back end (ABE) module 

25 132, which contains functionality for accessing annotation stores 17 and 18, for 
composing outgoing email based on annotation data, and for processing incoming 
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email. Incoming email to annotation server 10 is received and passed to the ABE 
module 132 by the Email Reply Server 133. If the incoming email includes the 
necessary information to generate an annotation, email reply server 133 converts the 
email into an annotation and forwards the annotation to ABE 132 for adding to 
5 annotation stores 17 and 18. Annotation content authored at client 15, using user 
interface 150, is received by ABE 132 and maintained in annotation content store 
17. Received meta data (control information) corresponding to the annotation 

content is maintained in annotation meta data store 18. 

> 

The annotation content and meta data can be stored in any of a variety of 
10 conventional manners, such as in SQL relational databases (e.g., using Microsoft 
"SQL Server" version 7.0, available from Microsoft Corporation). Annotation 
server 10 is illustrated in Fig. 3 as maintaining the annotation content and 
associated control information (meta data) separately in two different stores. 
Alternatively, all of the annotation data (content and meta information) can be 
1 5 stored together in a single store, or content may be stored by another distinct storage 
system on the network 16 in Fig. 1, such as a file system, media server, email 
server, or other data store. 

Annotation Storage Structure 

20 Fig. 4 shows an exemplary structure for an annotation entry 180 that is 

maintained by annotation server 10 in annotation meta data store 18 of Fig. 3. In 
the illustrated embodiment, an annotation entry 180 includes an author field 182, a 
time range field 184, a time units field 186, a creation time field 188, a title field 
190, a content field 192, an identifier field 194, a related annotation identifier field 

25 196, a set identifier(s) field 198, a media content identifier field 200, and an 
arbitrary number of user-defined property fields 202. Each of fields 182-202 is a 
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collection of data which define a particular characteristic of annotation entry 180. 
Annotation entry 180 is maintained by annotation server 10 of Fig. 3 in annotation 
meta data store 18. Content field 192, as discussed in more detail below, includes a 
pointer to (or other identifier of) the annotation content, which in turn is stored in 
5 annotation content store 17. 

Author field 182 contains data identifying the user who created annotation 
entry 1 80 and who is therefore the author of the annotation's content. The author is 
identified by ABE 151 of Fig. 3 based on the user logged into client 15 at the time 
the annotation is created, or alternatively by ABE 132 based on the sender of an 

1 0 email message. 

Time range field 184 contains data representing "begin" and "end" times 
defining a segment of media timeline with which annotation entry 180 is associated. 
Time units field 186 contains data representing the units of time represented in time 
range field 184. Together, time range field 184 and time units field 186 identify the 

1 5 relative time range of the annotation represented by annotation entry 1 80. This 
relative time range corresponds to a particular segment of the media stream to 
which annotation entry 180 is associated. The begin and end times for the 
annotation are provided by the user via interface 150 of Fig. 3, or alternatively can 
be automatically or implicitly derived using a variety of audio and video signal 

20 processing techniques, such as sentence detection in audio streams or video object 
tracking. 

It should be noted that the time ranges for different annotations can overlap. 
Thus, for example, a first annotation may correspond to a segment ranging between 
the first and fourth minutes of a media stream, a second annotation may correspond 
25 to a segment ranging between the second and seventh minutes of the media stream, 
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and a third annotation may correspond to a segment ranging between the second 
and third minutes of the media stream. 

Alternatively, rather than using the presentation timeline of the media 
content, different media characteristics can be used to associate the annotation with 
5 a particular segment(s) of the media content. For example, annotations could be 
associated with (or "anchored" on) specific objects in the video content, or specific 
events in the audio content. 

Creation time field 188 contains data specifying the date and time at which 
annotation entry 180 is created. It should be noted that the time of creation of 

10 annotation entry 180 is absolute and is not relative to the video or audio content of 
the media stream to which annotation entry 180 is associated. Accordingly, a user 
can specify that annotations which are particularly old, e.g., created more than two 
weeks earlier, are not to be displayed. ABE 132 of Fig. 3 stores the creation time 
and date when the annotation is created. 

15 Title field 190 contains data representing a title by which the annotation 

represented by annotation entry 180 is identified. The title is generally determined 
by the user and the user enters the data representing the title using conventional and 
well known user interface techniques. The data can be as simple as ASCII text or 
as complex as HTML code which can include text having different fonts and type 

20 styles, graphics including wallpaper, motion video images, audio, and links to other 
multimedia documents. 

Content field 192 contains data representing the substantive content of the 
annotation as authored by the user. The actual data can be stored in content field 
192, or alternatively content field 192 may store a pointer to (or other indicator of) 

25 the content that is stored separately from the entry 180 itself. The user enters the 
data representing the content using conventional and well known user interface 
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techniques. The content added by the user in creating annotation entry 180 can 
include any one or more of text, graphics, video, audio, etc. or links thereto. In 
essence, content field 192 contains data representing the substantive content the 
user wishes to include with the presentation of the corresponding media stream at 
5 the relative time range represented by time range field 1 84 and time units field 1 86. 

Annotation identifier field 194 stores data that uniquely identifies annotation 
entry 180, while related annotation identifier field 196 stores data that uniquely 
identifies a related annotation. Annotation identifier field 194 can be used by other 
annotation entries to associate such other annotation entries with annotation entry 

10 180. In this way, threads of discussion can develop in which a second annotation 
responds to a first annotation, a third annotation responds to the second annotation 
and so on. By way of example, an identifier of the first annotation would be stored 
in related annotation identifier field 196 of the second annotation, an identifier of 
the second annotation would be stored in related annotation identifier field 196 of 

15 the third annotation, and so on. 

Set identifiers) field 198 stores data that identifies a particular one or more 
sets to which annotation entry 180 belongs. A media stream can have multiple sets 
of annotations, sets can span multiple media content, and a particular annotation can 
correspond to one or more of these sets. Which set(s) an annotation belongs to is 

20 identified by the author of the annotation. By way of example, a media stream 
corresponding to a lecture may include the following sets: "instructor's 
comments", "assistant's comments", "audio comments", "text comments", "student 
questions", and each student's personal comments. 

Media content identifier field 200 contains data that uniquely identifies 

25 particular multimedia content as the content to which annotation entry 180 
corresponds. Media content identifier 200 can identify a single media stream 
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(either an individual stream or a composite stream), or alternatively identify 
multiple different streams that are different versions of the same media content. 
Media content identifier 200 can identify media versions in a variety of different 
manners. According to one embodiment, the data represents a real-time transport 
protocol (RTP) address of the different media streams. An RTP address is a type of 
uniform resource locator (URL) by which multimedia documents can be identified 
in a network. According to an alternate embodiment, a unique identifier is assigned 
to the content rather than to the individual media streams. According to another 
alternate embodiment, a different unique identifier of the media streams could be 
created by annotation server 10 of Fig. 3 and assigned to the media streams. Such a 
unique identifier would also be used by streaming media server 1 1 of Fig. 1 to 
identify the media streams. According to another alternate embodiment, a uniform 
resource name (URN) such as those described by K. Sollins and L. Mosinter in 
"Functional Requirements for Uniform Resource Names," IETF RFC 1733 
(December 1994) could be used to identify the media stream. 

User-defined property fields 202 are one or more user-definable fields that 
allow users to customize the annotation system. Examples of such additional 
property fields include a "reference URL" property which contains the URL of a 
web page used as reference material for the content of the annotation; a "help URL" 
property containing the URL of a help page which can be accessed concerning the 
content of the annotation; a "view script" property containing JavaScript which is to 
be executed whenever the annotation is viewed; a "display type" property, which 
gives the client user interface information about how the annotation is to be 
displayed; etc. 

Fig. 5 illustrates implicit annotation collections for annotations maintained 
by annotation server 10 of Fig. 3. A collection of annotations refers to annotation 
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entries 1 80 of Fig. 4 that correspond to the same media stream(s), based on the 
media content identifier 200. Annotation entries 180 can be viewed conceptually as 
part of the same annotation collection if they have the same media content 
identifiers 200, even though the annotation entries may not be stored together by 
5 annotation server 10. 

Annotation database 206 includes two annotation collections 208 and 210. 
Annotation server 10 dynamically adds, deletes, and modifies annotation entries in 
annotation database 206 based on commands from client 15. Annotation entries 
can be created and added to annotation database 206 at any time a user cares to 

10 comment upon the content of the stream (or another annotation) in the form of an 
annotation. Annotation server 10 forms an annotation entry from identification 
data, content data, title data, and author data of an "add annotation" request 
received from the client's ABE 151 (Fig. 3), and adds the annotation entry to 
annotation database 206. 

15 Annotation database 206 includes a fields 212, 214, and 216 that specify 

common characteristics of all annotation entries of database 206 or an annotation 
collection 208 or 210. Alternatively, fields 212-216 can be included redundantly in 
each annotation entry 1 80. 

Creator field 212 contains data identifying the user who was responsible for 

20 creating annotation database 206. 

RTP address fields 214 and 216 contains data representing an RTP address 
of the media content (e.g., the RTP addresses of each of the different streams 
contained in version table 204) for the annotation collection. An RTP address 
provides an alternative mechanism, in addition to the data in identifier field 200 of 

25 Fig. 4, for associating the media content with annotation entries 180. In alternative 
embodiments, RTP address fields 214 and 216 need not be included. 
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Annotation Notification and Creation 

An annotation can be created by a user of any of the client computers 15 of 
Fig. 1. As discussed above with reference to Fig. 3, client 15 includes an interface 
5 module 150 that presents an interface to a user (e.g., a graphical user interface), 
allowing a user to make requests of annotation server 10. In the illustrated 
embodiment, a user can access annotation server 10 via an annotation toolbar 
provided by interface 1 50. 

Fig. 6 illustrates an annotation toolbar in accordance with one embodiment 
10 of the invention. Annotation toolbar 240 includes various identifying information 
and user-selectable options 242-254. 

Selection of an exit or "X" button 242 causes interface 150 to terminate 
display of the toolbar 240. A server identifier 244 identifies the annotation server 
with which client 15 is currently configured to communicate (annotation server 10 
1 5 of Fig. 1 . in the illustrated embodiment). 

Selection of a connection button 246 causes ABE 1 5 1 of Fig. 3 to establish a 
connection with the annotation server identified by identifier 244. Selection of a 
query button 248 causes interface module 1 50 to open a "query" dialog box, from 
which a user can search for particular annotations. Selection of an add button 250 
20 causes interface module 150 to open an "add new annotation" dialog box, from 
which a user can create a new annotation. 

Selection of a show annotations button 252 causes interface module 150 to 
open a "view annotations" dialog box, from which a user can select particular 
annotations for presentation. 
25 Selection of a preferences button 254 causes interface 150 of Fig. 3 to open a 

"preferences" dialog box, from which a user can specify various UI preferences, 
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such as an automatic server query refresh interval, or default query criteria values to 
be persisted between sessions. 

Fig. 7 shows an exemplary "add new annotation" dialog box 260 that allows 
a user to create a new annotation. Dialog box 260 can be presented, for example, in 
5 response to user selection of add button 250 of Fig. 6. 

Dialog box 260 includes an annotation set identifier 262, a subject line 264, 
and an email field 266. Annotation set identifier 262 allows the user to identify a 
named set to which the new annotation will belong. This set can be a previously 
defined set, or a new set being created by the user. Selection of the particular set 

10 can be made from a drop-down menu activated by selection of icon 268, or 
alternatively can be directly input by the user (e.g., typed in using an alphanumeric 
keyboard). According to one embodiment of the invention, annotation server 10 of 
Fig. 3 supports read and write access controls, allowing the creator of the set to 
identify which users are able to read and/or write to the annotation set. In this 

15 embodiment, only those sets for which the user has write access can be entered as 
set identifier 262. Alternatively, rather than (or in addition to) user selection of an 
annotation set, one or more annotation sets may be inherently associated with dialog 
box 260 (or toolbar 240 of Fig. 6) and thus implicitly selected for each new 
annotation. 

* 20 Subject line 264 allows the user to provide a short summary or title of the 

annotation content. Although the subject line is illustrated as being text, it could 
include any of a wide variety of characters, alphanumerics, graphics, etc. In the 
illustrated embodiment, subject line 264 is stored in the title field 190 of the 
annotation entry of Fig. 4. 
25 Email field 266 allows the user to input the email address of a recipient of 

the annotation. The email addresses indicated in field 266 are provided to ABE 132 
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of annotation server 10, which in turn generates an email message including the 
newly created annotation and sends the email message to the recipient(s) indicated 
in field 266 in addition to adding the annotation to the annotation databases. 
Alternatively, the email message can be sent directly by client 15 of Fig. 1, or by an 
5 intermediary server (such as email server 13 of Fig. 1). 

Entries into email field 266 can be made on an individual or a "group" basis. 
Individual email addresses can be entered directly by the user, or can be selected 
from a pre-defined list. A list of available email addresses can be made available in 
any of a variety of conventional manners, such as via a pull-down or drop-down 

10 menu, a menu bar option, etc. A pre-defined list of email addresses can be obtained 
from a variety of different sources. For example, email agent 1 54 of Fig. 3 may 
have an "address book" or similar listing of email addresses programmed in by the 
user or other individuals. 

In one implementation, user interface 1 50 of Fig. 1 can have a list of email 

15 addresses associated with it, and/or the annotation set identified by set identifier 
262 may have a list of email addresses associated with it. These email addresses 
associated with either the user interface or the annotation set can be automatically 
entered as "default" email addresses, or alternatively can be individually selected by 
the user (e.g., via a pull-down or drop-down menu). 

20 Additionally, a group identifier can be entered into email field 266 to identify 

a group of email addresses. When the user adds an annotation with a group 
identifier in email field 266, annotation backend 132 of Fig. 3 (or alternatively 
another component, such as email server 13 of Fig. 1 or annotation backend 151 of 
Fig. 3) forwards an email message of the new annotation to each email address that 

25 corresponds to the group identifier. Group identifiers can be defined to correspond 
to email addresses in any of a variety of manners. For example, user interface 150 
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or the annotation set identified by set identifier 262 may have pre-defined groups 
associated with them, email agent 154 of Fig. 3 may have pre-defined groups 
associated with it, etc. 

Dialog box 260 further includes an annotation type selector in the form of 
5 radio buttons 270, 272, and 274, selection of which creates text annotation, an audio 
annotation, or a uniform resource locator (URL) annotation, respectively. Although 
not shown, other types of annotations could also be accommodated, such as 
graphics, video, etc. The content of the annotation is input to annotation field 276, 
which is text in the illustrated example of Fig. 7. Additional input controls (not 

10 shown) may also be included depending on the type of annotation being created. 
For example, conventional audio control buttons (such as fast forward, rewind, 
play, pause, stop, and record) may be included for audio annotations. 

Dialog box 260 also includes a checkbox 278 that allows the user to create 
an anonymous annotation. When checkbox 278 is selected (as illustrated in Fig. 7), 

15 the author of the annotation is not stored in author field 182 of Fig. 4. Annotation 
backend 151 can simply not forward the author information to annotation server 10 
of Fig. 3, or alternatively annotation server 10 may simply ignore the author 
information. 

Interface 150 assumes that the current media stream being presented to the 
20 user is the media stream to which the new annotation will be associated. The media 
stream to which an annotation is associated is referred to as the "target" of the 
annotation. An identifier of this stream is displayed in a target specification area 
280 of dialog box 260. Alternatively, a user could change the target of the 
annotation, such as by typing in a new identifier in target area 280, or by selection 
25 of a "browse" button (not shown) that allows the user to browse through different 
directories of media streams. 
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A time strip 282 is also provided as part of dialog box 260. Time strip 282 
represents the entire presentation time of the corresponding media stream. A 
"thumb" 283 that moves within time strip 282 indicates a particular temporal 
position within the media stream. The annotation being created via dialog box 260 
5 has a begin time and an end time, which together define a particular segment of the 
media stream. This segment is illustrated as darkened portion 284 of time strip 282. 
When "from" button 286 is selected, thumb 283 represents the begin time for the 
segment relative to the media stream. When "to" button 287 is selected, thumb 283 
represents the end time for the segment relative to the media stream. Alternatively, 
10 two different time bars could be displayed, one for the begin time and one for the 
end time. The begin and end times are also displayed in an hours/minutes/seconds 
format in boxes 288 and 289, respectively. 

Thumb 283 can be moved along time strip 282 in any of a variety of 
conventional manners. For example, a user can depress a button of a mouse (or 
1 5 other cursor control device) while a pointer is "on top" of thumb 283 and move the 
pointer along time strip 282, causing thumb 283 to move along with the pointer. 
The appropriate begin or end time is then set when the mouse button is released. 
Alternatively, the begin and end times can be set by entering (e.g., via an 
alphanumeric keyboard) particular times in boxes 288 and 289. 
20 Dialog box 260 also includes a "play" button 290. Selection of play button 

290 causes interface module 150 of Fig. 3 to forward a segment specification to 
web browser 153 of client 15. The segment specification includes the target 
identifier from target display 280 and the begin and end times from boxes 288 and 
289, respectively. Upon receipt of the segment specification from interface module 
25 150, the browser communicates with media server 11 and requests the identified 
media segment using conventional HTTP requests. In response, media server 1 1 
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streams the media segment to client 15 for presentation to the user. This 
presentation allows, for example, the user to verify the portion of the media stream 
to which his or her annotation will correspond. 

Dialog box 260 also includes a cancel button 292, an OK button 294, and an 
5 advanced button 296. Selection of cancel button 292 causes interface 150 to close 
the dialog box 260, canceling out of the add new annotation process. Selection of 
OK button 294 causes interface 150 to forward the information entered in dialog 
box 260 to annotation backend 151, which sends the new annotation information to 
annotation server 10 of Fig. 3. Interface 150 then closes dialog box 260, the new 

10 annotation being entered. 

Selection of "advanced" button 296 reduces the number of options available 
to the user, simplifying dialog box 260. For example, the simplified dialog box may 
not include fields and buttons 280 - 290. 

As discussed above, when a new annotation is created, an email message is 

15 created and sent to each email address identified in email field 266. The email 
message is a notification to each of the email recipients (as identified by the email 
addresses/groups in field 266) of the new annotation. Annotation back end 151 of 
client 15 includes various identifying information for the annotation and its 
corresponding media content. Some or all of this identifying information may be 

20 made visible to the user, or alternatively may be embedded in header or footer 
sections of the message that are not displayed to the user. 

Fig. 8 shows an example email message notification of a new annotation. 
Email interface 155 of Fig. 3 provides various conventional user-selectable options 
300 for viewing, creating, and manipulating electronic mail messages. Email 

25 interface 155 also displays email message 302 that includes the content 304 of the 
new annotation (e.g., the text, audio, video, etc.), which is text in the illustrated 
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example. Thus, the recipient of email message 302 can play back or view the new 
annotation content 304 without having to access the annotation server or the 
underlying media content. Email message 302 also includes various identifiers and 
descriptive information regarding the new annotation and the multimedia content to 
5 which it corresponds, including a host identifier 306, a target identifier 308, an 
annotation identifier 310, and an annotation set identifier 312. 

Host identifier 306 (such as a uniform resource locator) identifies the host of 
the media content and a temporal segment of the media content to which the new 
annotation corresponds. Host identifier 306 is an identifier of the host of the media 

10 content (e.g., a URL of a web page that displays the media content) concatenated 
with time range information, and is presented as a user-selectable link to the media 
content (e.g., a text or graphic hyperlink). The identifier displayed to the user can 
be the URL of the media content, or alternatively some other text, graphic, video, 
etc. identifier of the media content. If the user selects the identifier, email agent 

15 154 of Fig. 3 forwards host identifier 306 to annotation back end 151. Annotation 
back end 151 forwards host identifier 306 to web browser 153 which in turn 
accesses streaming media server 1 1 of Fig. 1 to request the media content and begin 
playback of the media content to the user. Annotation back end can also access 
annotation server 10, using host identifier 306, to obtain additional annotations 

20 corresponding to the media content. Alternatively, email agent 154 may forward 
host identifier 306 directly to web browser 153. 

Host identifier 306 also identifies the temporal range or segment of the 
media content with which the new annotation is associated. This temporal range is 
the beginning and end time for the annotation, displayed in boxes 288 and 289 of 

25 Fig. 7. When a user selects host identifier 306 in email message 302, the temporal 
range information is forwarded to media server 1 1 . Media server 1 1 then begins 
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streaming the media content to client 15, starting with the temporal location 
identified by the begin time in host identifier 306. Media server 11 can then 
continue to stream the remainder of the media content to client 15, or alternatively 
may stream only the segment identified by the temporal range identifier (defined by 
5 the begin and end time for the annotation). 

Target identifier 308 (such as a uniform resource locator) identifies the entire 
media content that the new annotation corresponds to. Target identifier 308 is the 
target identifier from target display 280 of Fig. 7, and is presented as a user- 
selectable link to the media content (e.g., a text or graphic hyperlink). If the user 

10 selects target identifier 308, email agent 154 of Fig. 3 forwards the URL of the 
media content to annotation back end 151, which in turn accesses streaming media 
server 1 1 of Fig. 1, as well as possibly annotation server 10 and web page server 12, 
to request the media content and begin playback of the media content to the user. 

Annotation identifier 310 uniquely identifies the new annotation. Identifier 

15 310 is assigned by the annotation back end 132 of Fig. 3. Annotation identifier 310 
can be used by annotation server 10 if the user selects to create another new 
annotation by simply "replying" to the email message, as discussed in more detail 
below. Annotation identifier 310 informs annotation server 10 of which annotation 
the "reply" annotation corresponds to. 

20 Annotation set identifier 312 indicates the annotation set(s) to which the new 

annotation corresponds. Identifier 312 can be used by annotation server 10 if the 
user selects to create another new annotation by replying to email message 302. 
Identifier 312 informs annotation server 10 which annotation set(s) the "reply" 
annotation corresponds to. 

25 Email message 302 also includes an installation option 314. Situations can 

arise where the recipient of email message 302 does not have the necessary modules 
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(e.g., interface 150 and annotation backend 151 of Fig. 3) to present annotations to 
a user concurrent with the corresponding multimedia content. Installation option 
314 is a user-selectable link (e.g., a text or graphic hyperlink), that identifies an 
installation program (e.g., at annotation server 10). If the user selects installation 
5 option 314, email agent 154 of Fig. 3 forwards the selected URL to web browser 
153, which in turn access the appropriate server (e.g., annotation server 10). The 
installation program can then run, installing the appropriate modules on the client 
computer 15. 

In addition to using email message 302 to play back annotations and/or 

10 corresponding media content, the recipient of electronic mail message 302 can also 
create an additional annotation by "replying" to message 302. This "reply" is 
generated by user selection of the conventional "reply to sender" or "reply" option 
provided by email interface 155, which is typically available in conventional email 
user interfaces, such as "reply" button 316. When the user selects to reply, the 

15 sender 318 of the original email message (annotation server 10) is entered as the 
recipient of the reply message. The user can then enter in a new annotation via the 
email interface 155 (which can be text, audio, video, or any media format supported 
by the email interface 155 and agent 154). When the user finishes entering the 
additional annotation, the user selects the "send" option which forwards the reply 

20 message to annotation server 10. Annotation server 10 stores this additional 
annotation, creating an annotation entry 180 of Fig. 3 for the new annotation. In the 
illustrated embodiment, this new annotation identifies, in the related annotation 
identifier field 196 of Fig 5, the original annotation that was created when the 
original email message was sent by annotation server 10. 

25 Fig. 9 shows an example email reply message including a new annotation. 

Email reply message 332 includes the content 304 of the original annotation (e.g., 
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the text, audio, video, etc.), as well as the identifiers 306-314 of the original 
annotation. Reply message 332 further includes additional new content 334 for 
another new annotation. When the user selects "send" button 336, the reply 
message 332 is forwarded to annotation server 10. 
5 Upon receipt of electronic mail reply message 332, annotation server 10 

creates another new annotation and uses the reply message content 334 as the 
content for this new annotation. The author of the new annotation is the sender of 
reply message 332. Annotation server 10 uses identifiers 306-312 describing the 
original annotation (from email message 302 of Fig. 8) as needed to create a new 

10 annotation entry 180 of Fig. 4. Annotation server 10 associates this new annotation 
with the same media content as the annotation in the original email (as identified by 
target identifier 308 and host identifier 306 of Fig. 8). Annotation server 10 also 
associates this new annotation with the same temporal range as the annotation in the 
original email (as identified by host identifier 306 of Fig. 8). Annotation server 10 

15 further identifies, as a related annotation, the original annotation that was created 
when the original mail message was sent by annotation server 10 (as identified by 
annotation identifier 310 of Fig. 8). Annotation server 10 also associates the new 
annotation with the same annotation set(s) as the original annotation message (as 
identified by annotation set identifier 312 of Fig. 8). 

20 Alternatively, annotation server 10 may obtain the annotation identifier of 

the original annotation from the reply message 332 and access the annotation entry 
180 of Fig. 4 with that identifier. Information from that annotation entry is then 
used to create the new entry (e.g., fields 184-190, 196, and 198 of Fig. 4 are copied 
to the new annotation entry). 

25 In the illustrated examples of Figs. 8 and 9, various identifiers 306-314 are 

presented to the user. Alternatively, one or more of these identifiers may be 
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included in the email messages but hidden from view. For example, one or more 
"hidden" or "invisible" properties or parameters may be included in the email 
message which are accessible by email agent 1 54 but are not displayed to the user. 

Fig. 10 is a flowchart illustrating exemplary steps in accessing media content 
5 from an email message in accordance with the invention. The steps on the left side 
of Fig. 10 are implemented by client computer 15 of Fig. 1, and the steps on the 
right side of Fig. 10 are implemented by streaming media server computer 11. The 
steps of Fig. 10, on both client and server computers, may be performed in 
software. Fig. 10 is described with additional reference to components in Fig. 1. 

10 An email message notification of a new annotation is received at the client 

(step 352). This email message notification of the new annotation includes the 
content of the new annotation, an URL of the corresponding media content, a 
temporal range identifier for the new annotation, and an identifier of the annotation. 
The client displays the email message via email interface 155, including a user- 

1 5 selectable identifier of the media content (step 354). 

In the illustrated example of Fig. 10, assume that the user selects the 
identifier of the media content. The user-selection of the identifier is received by 
the client (step 356), which in turn sends a request for the media content to the 
streaming media server (step 358). The request for media content includes the 

20 temporal range identifier that identifies the segment of the media content that the 
annotation corresponds. 

The request for media content, including temporal range information, is 
received by the media server (step 360). The media server then begins streaming 
the requested media content to the client, beginning with the starting point of the 

25 segment identified by the temporal range information (step 362). The streaming 
media content is received by and presented by the client (step 364). 
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In addition, other types of email messages that are not direct replies to an 
email notification message can be used to add new annotations. Typical email 
applications provide a user with a "forward" option that allows the user to send a 
received email message to one or more other users, optionally appending an 
5 additional message by the user. Such email messages can be forwarded from user 
to user, creating a string of email messages often referred to as an "email thread". 
Such forwarding can be found in numerous settings, such as system designers that 
carry on "conversations" or discuss various options with one another via such email 
messages. Thus, an original annotation notification email message can be 

10 forwarded around to multiple additional users. The annotation server, however, is 
not aware of such additional messages in an email thread unless the annotation 
server is listed as a recipient of the messages (due to either replying to or 
forwarding of a message). If one of the users forwarding the email thread, 
however, were to forward the thread to the annotation server, then all of the email 

1 5 messages in the thread so far could be added as annotations. 

Fig. 11 is an exemplary diagram illustrating the creation of an email thread. 
Initially, an email notification message 372 of a new annotation is sent from the 
annotation server to user A. Upon receipt of the message 372, user A comments on 
the new annotation and forwards it to user B. User B thus receives an email 

20 message 374 including the content of the original email notification message 372 as 
well as the additional content 376 added by user A. User B similarly comments on 
the new annotation and the comments of user A and forwards the email message to 
User C. User C thus receives an email message 378 including the content of the 
original email notification message 372, the additional content 376 added by user 

25 A, and the additional content 380 added by user B. User C can similarly add 
additional content to the email message thread, and forward a message 382 
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including the original email notification message 372, the additional content 376 
added by user A, the additional content 380 added by user B, and the additional 
content 384 added by user C. 

Upon receipt of the email message 382, user A can decide to forward the 
5 message 382 to annotation server 10 of Fig. 3. By forwarding the message 382, 
which is an email thread including multiple email messages from multiple users, 
annotation server 1 0 can add the email messages from the thread into one or more 
annotations. Thus, even though some of the email messages were not forwarded to 
annotation server 10, they can subsequently be added to annotations. 

10 When annotation server 10 receives an email thread such as message 382, it 

generates one or more annotations based on the identifying information in the 
thread. As the original email message 372 was an annotation notification email 
message, message 372 includes identifying information for the original annotation, 
including media content identifier and temporal range identifier. This identifying 

1 5 information is used for all additional annotations that are created from message 382. 

Annotation server 10 analyzes the message 382 and determines the portion 
that was not part of the original annotation notification email message 372, and 
creates an additional annotation for this portion. Alternatively, annotation server 10 
may use all of message 382 to create a single annotation. In another alternative 

20 implementation, annotation server analyzes the message 382 to identify each 
individual addition to the original email notification message (based, for example, 
on header information in each message) and creates a separate annotation for each 
addition. Other alternative implementations may also be used, such as grouping 
additions by the same user into the same annotation, etc. 
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The forwarding of messages illustrated in Fig. 1 1 can occur with any time 
delays between forwardings. For example, user A may forward message 374 to 
user B two minutes or two years after receiving the email notification message 372. 

Additionally, a user may generate new annotations from email messages that 
5 are not a reply to or forwarding of another email message. In order to generate such 
an annotation, sufficient information needs to be included in the email message to 
allow annotation server 10 to generate the annotation. Specifically, identifying 
information of the media content to which the annotation corresponds needs to be 
identified to annotation server 10. Additional information may also optionally be 
1 0 required, such as a temporal range identification for the annotation and annotation 
set identification. 

This information can be included in the email message in a variety of 
manners. In one implementation, the information is manually put into the message 
by the user, such as by input from a keyboard or by using a "cut and paste" 

15 methodology. Alternatively, web page server 12 of Fig. 1 may provide the 
multimedia player interface of Fig. 13, which may display as part of the interface 
the necessary identifying information. One selection of the multimedia player 
interface may be a "new annotation" form that can be selected by the user. The web 
page server 12 includes, visible or invisible to the user, the necessary identifying 

20 information for annotation server 10 to generate a new annotation. 

In another alternative implementation, the email agent 1 54 of Fig. 3 may 
communicate with the web page server 12 or streaming media server 1 1 to obtain 
the necessary identifying information for annotation server 10 to generate the new 
annotation. 

25 Fig. 12 is a flowchart illustrating an exemplary process in adding new 

annotations from an email message in accordance with the invention. The process 
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is implemented by annotation server 10 of Fig. 3, and may be performed in 
software. Fig. 12 is described with additional reference to components in Fig. 3. 

Initially, the annotation server receives an email message with annotation 
content (step 402). The annotation server analyzes the email message and locates 
5 the annotation data in the email message (step 404), generating at least one new 
annotation from the annotation data in the email message (step 406). The number 
of annotations generated can vary (e.g., depending on whether the email message is 
an email thread). 

The annotation server also locates identifying information for media content 
10 corresponding to the new annotation(s) (step 408). The identifying information 
includes both the media content identifier and the temporal range identifier. This 
identifying information is used for each new annotation created. The annotation 
server then adds these new annotation(s) to the annotation store (step 410). 

15 Annotation and Media Segment Retrieval 

Fig. 13 shows one implementation of a graphical user interface window 500 
that concurrently displays annotations and corresponding media segments. This UI 
window 500 has an annotation screen 454, a media screen 456, and a toolbar 240. 

Media screen 456 is the region of the UI within which the multimedia 
20 content is rendered. For video content, the video is displayed on screen 456. For 
non-visual content, screen 456 displays static or dynamic images representing the 
content. For audio content, for example, a dynamically changing frequency wave 
that represents an audio signal could be displayed in media screen 456. 

Annotation screen 454 is the region of the UI within which the annotations 
25 are rendered. For video, graphical, and text annotations, the video, graphical, or 
text content of the annotation is displayed on screen 454. For non-visual content, 
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screen 454 displays static or dynamic images representing the annotation content, 
such as the title or summary of the annotation, or a dynamically changing frequency 
wave in the case of audio content. 



5 Conclusion 

The invention described above provides for the creation and notification of 
new annotations via electronic mail (email). A user, in creating a new annotation, 
can identify one or more other users that are to receive an email notification of the 
new annotation. Upon receiving such an email notification, the recipient(s) can 

10 advantageously access, via a single user-selectable link, the temporal segment of 
multimedia content corresponding to the new annotation. Furthermore, additional 
annotations can advantageously be generated from email, by replying to an email 
notification of a new annotation, by sending a new email message or by forwarding 
an email message or email thread, etc. 

15 Although the invention has been described in language specific to structural 

features and/or methodological steps, it is to be understood that the invention 
defined in the appended claims is not necessarily limited to the specific features or 
steps described. Rather, the specific features and steps are disclosed as preferred 
forms of implementing the claimed invention. 

20 
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CLAIMS 

1. A system comprising: 

a client computer to playback multimedia content and annotations 
5 corresponding to different temporal portions of the multimedia content; and 
an annotation server, coupled to the client computer, to, 

maintain an annotation database having a plurality of annotations 
corresponding to the multimedia content, 

provide the plurality of annotations to the client computer for 
10 playback, 

send electronic mail messages including annotations to recipients 
identified by the client computer, 

generate new annotations based on received electronic mail messages, 

and 

15 add the new annotations to the annotation database. 

2. A system as recited in claim 1 , further comprising a media server to 
manage streaming the multimedia content to the client computer. 

20 3. A system as recited in claim 1, further comprising a network coupling 

the client computer to the annotation server. 

4. A system as recited in claim 1, wherein the client computer is further 

to: 

25 present an electronic mail message including a multimedia content 

identifier to a user, 
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receive a user selection of the multimedia content identifier, 
access a media server to obtain the multimedia content, and 
playback the multimedia content to the user. 

5 5. A system as recited in claim 4, wherein the client computer is further 

to transmit, to the media server, an identifier of a temporal segment of the 
multimedia content, and wherein the media server is to stream to the client 
computer the multimedia content beginning with the identified temporal segment. 

10 6. A method comprising: 

presenting, to a user, a user interface allowing the user to create a new 
annotation corresponding to media content; and 

including, as part of the user interface, a field via which the user can identify 
a recipient that is to receive an electronic mail notification of the new annotation. 

15 

7. A method as recited in claim 6, further comprising forwarding the new 
annotation to an annotation server to transmit the electronic mail notification of the 
new annotation to the recipient. 

20 8. A method as recited in claim 6, further comprising forwarding the new 

annotation to an annotation server to be added to an annotation set corresponding to 
the media content. 

9. A method as recited in claim 6, further comprising: 
25 allowing the user to identify, via the field of the user interface, a plurality of 

recipients to receive the electronic mail notification; and 
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wherein the forwarding comprises forwarding, to each of the plurality of 
recipients, the electronic mail notification of the new annotation. 

10. A method as recited in claim 6, further comprising: 

5 allowing the user to input, via the field of the user interface, a group 

identifier; 

identifying an electronic mail address for each of a plurality of recipients 
corresponding to the group identifier; and 

wherein the forwarding comprises forwarding, to each of the plurality of 
10 recipients, the electronic mail notification of the new annotation. 

11. A method as recited in claim 6, further comprising: 

presenting, as part of the user interface, a default set of recipients to receive 
the electronic mail notification. 

15 

12. One or more computer-readable memories containing a computer 
program that is executable by a processor to perform the method recited in claim 6. 

13. A method comprising: 

20 receiving data for a new annotation corresponding to a temporal range of 

media content; 

generating an electronic mail message including both the content of the new 
annotation and an identifier of the media content; and 

forwarding the electronic mail message to a recipient identified by the data. 

25 
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14. A method as recited in claim 13, wherein the generating further 

comprises including, in the electronic mail message, an identifier of the new 
annotation. 



5 15, A method as recited in claim 13, wherein the generating further 

comprises including, in the electronic mail message, an identifier of the temporal 
range of media content. 

16. One or more computer-readable memories containing a computer 
10 program that is executable by a processor to perform the method recited in claim 

13. 

17. A method comprising: 

receiving an electronic mail notification of a new annotation corresponding 
1 5 to media content, the media content having a plurality of temporal segments; 
presenting the electronic mail notification to a user; 
receiving a user input to access the media content; and 

accessing, upon receiving the user input, a media server to stream one of the 
plurality of segments that corresponds to the new annotation to the user. 

20 

18. A method as recited in claim 17, wherein the presenting comprises 
displaying annotation content for the new annotation to the user. 

19. A method as recited in claim 1 7, wherein the presenting comprises: 
25 displaying annotation content for the new annotation to the user; and 
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displaying at least one identifier to the user, the at least one identifier 

including one or more of: 

a content identifier that identifies the media content; 
a range identifier that identifies a temporal range of the segment 
corresponding to the new annotation; 

an annotation identifier that identifies the new annotation; and 
an annotation set identifier that identifies one or more annotation sets 
that the new annotation is part of. 

20. A method as recited in claim 19, wherein the content identifier 
comprises a uniform resource locator (URL). 



21. A method as recited in claim 17, wherein the receiving a user input 
comprises receiving a user selection of a uniform resource locator (URL) of the 

1 5 media content. 

22. A method as recited in claim 17, wherein the receiving a user input 
comprises receiving a user selection of an identifier of the annotation. 



20 



23. One or more computer-readable media having stored thereon a 
computer program that, when executed by one or more processors, causes the one 
or more processors to perform functions including: 

receiving an electronic mail notification of a new annotation corresponding 
to media content, wherein the electronic mail notification includes a user-selectable 
25 identifier of the media content; 

displaying the electronic mail notification; 
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receiving a user selection of the identifier; and 

accessing, upon receiving the user selection, a media server indicated by the 
identifier to begin streaming the media content for presentation to the user. 

5 24. One or more computer-readable media as recited in claim 23, wherein 

the user-selectable identifier comprises a uniform resource locator (URL) that 
identifies a server and a location at the server where the media content is located. 

25. A system comprising: 

an interface module to receive data regarding a new annotation 
corresponding to media content; 

a module to generate an electronic mail message regarding the new 
annotation, the electronic mail message including, 
the new annotation content, and 

an identifier of the media content to which the new annotation 
corresponds. 

26. A system as recited in claim 25, wherein the electronic mail message 
further includes an identifier of the temporal range of the media content that the 

20 new annotation is associated with. 

27. A system as recited in claim 25, wherein the electronic mail message 
further includes a unique identifier of the new annotation. 



10 



15 
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28. A system as recited in claim 25, wherein the electronic mail message 
further includes an identifier of one or more annotation sets that the new annotation 
is associated with. 

5 29. A system as recited in claim 25, wherein the system comprises an 

annotation server computer. 

30. A system as recited in claim 25, wherein the system comprises a 
client computer and wherein the interface module comprises a user interface. 

10 

31. A method comprising: 

receiving an electronic mail notification of an annotation corresponding to 
media content; and 

replying to the electronic mail notification to generate a new annotation 
1 5 corresponding to the media content. 

32. A method as recited in claim 3 1 , wherein the replying includes: 
obtaining, from the electronic mail notification, an identifier of the 

annotation; and 

20 including the identifier of the annotation in the reply. 

33. A method comprising: 

receiving an electronic mail thread including a plurality of electronic mail 
messages; 

25 creating an annotation from at least one of the electronic mail messages; and 

adding the annotation to an annotation database. 
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34. A method as recited in claim 33, wherein the creating comprises 
creating a single annotation including each of the plurality of electronic mail 
messages. 

35. A method as recited in claim 33, wherein: 

the creating comprises generating, for each of the plurality of electronic mail 
messages, an annotation; and 

the adding comprises adding each of the generated annotations to the 
annotation database. 

36. A method as recited in claim 33, further comprising locating, in the 
electronic mail thread, an identifier of media content that the annotation 
corresponds to. 

15 

37. One or more computer-readable media having stored thereon a 
computer program that, when executed by one or more processors, causes the one 
or more processors to perform functions including: 

receiving an electronic mail message; and 
20 extracting both annotation content and annotation identification information 

from the electronic mail message. 



5 



10 
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38. One or more computer-readable media as recited in claim 37, wherein 
the computer program further causes the one or more processors to perform 
functions including: 

creating a new annotation based on the extracted annotation content and the 
5 annotation identification information; and 

adding the new annotation to an annotation database. 

39. One or more computer- readable media as recited in claim 37, wherein 
the extracting annotation identification information comprises extracting from the 

10 email message: 

an identifier of media content to which the annotation content corresponds 
and of a temporal segment, corresponding to the annotation content, of the media 
content; and 

an identifier of an annotation set that a new annotation including the 
15 extracted annotation content is to be part of 
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